Predictive Analytics and Machine Learning

Let us go back to basics and model the association between a set of explanatory variables (covariates, independent variables, right-hand-side-variables, \(X\)’s, etc …) and a response variable (dependent variable, left-hand-side-variable, or just simply …) \(Y\) using the following linear relationship:

\[Y = \beta_0 + \beta_1X_1 + \cdots + \beta_pX_p + \epsilon,\] where \(\epsilon\) is a random variable typically assumed to be normally distributed. If \(Y\) is a continuous variable we call this a regression problem.

If \(Y\) is a binary (dummy, zero-one) variable, we use the logistic regression:

\[\textrm{logit}(P(Y = 1|X_1,\ldots,X_p)) = \beta_0 + \beta_1X_1 + \cdots + \beta_pX_p.\] We call this a classification problem.

• In classical econometrics we are often interested in statistical and causal inference.
• In modern data science we are more interested in predictions, but this distinction should not be taken too far.
• We will focus on the classification problem in the examples today, because it is easier to visualize.

• The logistic regression is a classical econometric model designed for statistical inference, i.e. figuring out which explanatory variables contributes to variation in the “success probability” \(P(Y = 1)\), and in which direction.
• The logistic regression defines a proper probability model: \[\begin{equation} \log\left(\frac{P(Y = 1)}{1 - P(Y=1)}\right) = \beta_0 + \beta_1X_1 + \cdots + \beta_pX_p. \end{equation}\]
• This is classical parametric statistics. Unless you introduce polynomial (or otherwise curved) terms, the LR always leads to a straight line decision boundary.
• the LR is interpretable, requires less data (because we have decided the shape of the boundary even before we see any data).
• But if we
  1. don’t want to assume that this probability model is (approximately) true, and
  2. are more interested in predicting the status of new customers than interpreting the estimated model,
  3. (and have a reasonable amount of data),
• we might want to do something else!

\(k\) nearest neighbours algorithm for predicting the class of a new individual:

• Select an integer \(k\).
• Identify the \(k\) individuals in the training data which is closest/most similar to the new individual.
• Perform a “majority vote” among these individuals in order to predict the status of the new individual.

• \(k = 1\) gives a perfect prediction on the training data, but horrible out-of-sample predictions – the model is overfitted.
• \(k = n\) results in total ignorance of the explanatory variables. You always predict into the majority class in the training data – it is underfitted.
• The best value must lie somewhere in between, but where, and how do we find it?

• If we use the same data for estimating (training) the model and choosing the complexity we make the classical overfitting mistake.
• In the kNN case, we will always end up with \(k=1\), because that gives perfect predictions on the training data.
• It will result in catastrophic predictions, however.
• We can fix this by splitting the data randomly into three parts:

• The three parts of the data set play different roles when estimating the model:
  • The training data is used to estimate/train the model.
  • We use the validation data to tune the model: Which value of \(k\), when used to estimate the model using the training data, gives the best predictions on the validations set?
  • We use the testing data to test the model afterwards: how well does the model predict unseen data (out-of-sample)?.
• It is abslolutely crucial that the testing data is kept separate from model training and tuning. Checking predictions on data that was used to train a model is considered cheating!

We can measure the performance of prediction models in many ways.

• Making a confusion matrix:

• Drawing the reciever operating characteristic (ROC), takes all classification thresholds into account: